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(54) DATA PROCESSORDATA PROCESSING METHOD AND MEDIUM 
(57)Abstract: 

PROBLEM TO BE SOLVED: To improve the processing performance of data. 
SOLUTION: For respective directions D1D2...D16 whose start point is a center 
pixelbased on the waveform characteristics (characteristics of pixel value) of an 
SD pixel present in the directionthe SD pixel for finally constituting a tap is 
decided. That isfor the respective directions Dibased on the waveform 
characteristics of an SD image present in the directionthe position of the SD pixel 
for constituting the tap present in the direction is moved. 



[Claim(s)] 

[Claim 1]A data processing device which processes input data and predicts output 
data to the input datacomprising: 

A determination means to determine two or more data extracted from said input 

data based on the waveform characteristic of said input data. 

An extraction means to extract two or more data from said input data according 

to determination in said determination means about attention output data which is 

said output data which is going to calculate a predicted value. 

A prediction means which calculates a predicted value of said attention output 

data based on two or more data extracted in said extraction means. 

[Claim 2]Based on two or more data extracted in said extraction meansclass 
sorting is performed about said attention output dataThe data processing device 



CLAIMS 



according to claim 1 wherein said prediction means calculates a predicted value of 
said attention output data using a predetermined prediction coefficient 
corresponding to said class codeincluding further a class sorting means to output 
a corresponding class code. 

[Claim 3]The data processing device according to claim 2wherein said prediction 
means linearity-primary-predicts said attention output data using said prediction 
coefficient. 

[Claim 4]The data processing device according to claim 3wherein said prediction 
means linearity-primary-predicts said attention output data using said prediction 
coefficient and two or more data extracted in said extraction means. 
[Claim 5]The data processing device according to claim 2 including further a 
prediction coefficient memory measure which has memorized said prediction 
coefficient for said every class code. 

[Claim 6]The data processing device according to claim 1 wherein said 
determination means detects an extremum of said input data and determines two 
or more data extracted from said input data based on the detection result. 
[Claim 7]The data processing device according to claim 1 wherein said 
determination means calculates a difference value of said input data and 
determines two or more data extracted from said input data based on the result of 
an operation. 

[Claim 8]The data processing device according to claim 1 wherein said 
determination means determines two or more data which calculates an error of 
said input data to a function which approximates said input dataand is extracted 
from said input data based on the result of an operation. 

[Claim 9]The data processing device according to claim 1 wherein said input data 
and output data are image data. 

[Claim 10]The data processing device according to claim 9wherein said extraction 
means extracts a pixel which exists on the outskirts spatially or in time from image 
data as said input data to a pixel as said attention output data. 
[Claim 1 1]A data processing method which processes input data and predicts 
output data to the input datacomprising: 

A determination step which determines two or more data extracted from said input 
data based on the waveform characteristic of said input data. 

An extraction step which extracts two or more data from said input data according 
to determination in said determination step about attention output data which is 
said output data which is going to calculate a predicted value. 
A prediction step which calculates a predicted value of said attention output data 
based on two or more data extracted in said extraction step. 

[Claim 12]A medium which makes a computer execute a program for performing 
data processing which processes input data and predicts output data to the input 
datacomprising: 

A determination step which determines two or more data extracted from said input 
data based on the waveform characteristic of said input data. 



An extraction step which extracts two or more data from said input data according 
to determination in said determination step about attention output data which is 
said output data which is going to calculate a predicted value. 
A prediction step which calculates a predicted value of said attention output data 
based on two or more data extracted in said extraction step. 

[Claim 13]A data processing device which learns a prediction coefficient used for 
predicting output data [ as opposed to / process input data and / the input data ] 
characterized by comprising the following. 

A creating means which generates student data which serves as a student from 

teacher data which serves as a teacher for study of said prediction coefficient. 

A determination means to determine two or more data extracted from said student 

data based on the waveform characteristic of said student data. 

An extraction means to extract two or more data from said student data according 

to determination in said determination means about attention teacher data which 

is said teacher data which is going to calculate a predicted value. 

A calculating means which calculates said prediction coefficient based on two or 

more data extracted in said extraction means. 

[Claim 14]Based on two or more data extracted in said extraction meansclass 
sorting is performed about said attention teacher dataThe data processing device 
according to claim 13 characterized by said calculating means containing ** in 
quest of said prediction coefficient using two or more data extracted in said 
extraction means for said every class code including further a class sorting means 
to output a corresponding class code. 

[Claim 15]The data processing device according to claim 13wherein said 
calculating means calculates said prediction coefficient for said teacher data to be 
obtained by linearity primary prediction. 

[Claim 16]The data processing device according to claim 15wherein said 
calculating means calculates said prediction coefficient for said teacher data to be 
obtained by linearity primary prediction using two or more data extracted in said 
extraction means. 

[Claim 17]The data processing device according to claim 13wherein said 
determination means detects an extremum of said student data and determines 
two or more data extracted from said student data based on the detection result. 
[Claim 18]The data processing device according to claim 13wherein said 
determination means calculates a difference value of said student data and 
determines two or more data extracted from said student data based on the result 
of an operation. 

[Claim 19]The data processing device according to claim 13wherein said 
determination means determines two or more data which calculates an error of 
said student data to a function which approximates said student dataand is 
extracted from said student data based on the result of an operation. 
[Claim 20]The data processing device according to claim 13wherein said student 



data and teacher data are image data. 

[Claim 21]The data processing device according to claim 20wherein said extraction 
means extracts a pixel which exists on the outskirts spatially or in time from image 
data as said student data to a pixel as said attention teacher data. 
[Claim 22]A data processing method which learns a prediction coefficient used for 
predicting output data [ as opposed to / process input data and / the input data ] 
characterized by comprising the following. 

A generation step which generates student data which serves as a student from 
teacher data which serves as a teacher for study of said prediction coefficient. 
A determination step which determines two or more data extracted from said 
student data based on the waveform characteristic of said student data. 
An extraction step which extracts two or more data from said student data 
according to determination in said determination step about attention teacher data 
which is said teacher data which is going to calculate a predicted value. 
An arithmetic step which calculates said prediction coefficient based on two or 
more data extracted in said extraction step. 

[Claim 23]A medium which makes a computer execute a program for performing 
data processing which learns a prediction coefficient used for predicting output 
data [ as opposed to / process input data and / the input data ] characterized by 
comprising the following. 

A generation step which generates student data which serves as a student from 
teacher data which serves as a teacher for study of said prediction coefficient. 
A determination step which determines two or more data extracted from said 
student data based on the waveform characteristic of said student data. 
An extraction step which extracts two or more data from said student data 
according to determination in said determination step about attention teacher data 
which is said teacher data which is going to calculate a predicted value. 
An arithmetic step which calculates said prediction coefficient based on two or 
more data extracted in said extraction step. 

[Claim 24]A data processing device which processes input data characterized by 
comprising the followingand is provided with the 1st device that predicts output 
data to the input dataand the 2nd device that learns a prediction coefficient used 
for predicting said output data. 

The 1st determination means that determines two or more data which extracts 
said 1st device from said input data based on the waveform characteristic of said 
input data. 

The 1st extraction means that extracts two or more 1st data from said input data 
according to determination in said 1st determination means about attention output 
data which is said output data which is going to calculate a predicted value. 
A creating means which generates student data which serves as a student from 
teacher data in which said 2nd device serves as a teacher for study of said 
prediction coefficient including a prediction means which calculates a predicted 



value of said attention output data based on two or more 1st data extracted in 
said 1 st extraction means. 

The 2nd determination means that determines two or more data extracted from 
said student data based on the waveform characteristic of said student dataThe 
2nd extraction means that extracts two or more 2nd data from said student data 
according to determination in said 2nd determination means about attention 
teacher data which is said teacher data which is going to calculate a predicted 
valueA calculating means which calculates said prediction coefficient based on two 
or more data extracted in said 2nd extraction means. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention]This invention relates to the data processing device it 
enables it to raise the processing performance at the time of performing data 
processingsuch as image dataa data processing methodand a mediumconcerning a 
data processing devicea data processing methodand a medium. 
[0002] 

[Description of the Prior Art]This applicant has proposed class sorting adaptation 
processing previouslyfor example as processing for improving improvement and 
other picturessuch as image quality of a picture. 

[0003]Class sorting adaptation processing consists of class sorting processing and 
adaptive processingand performs adaptive processing for data for every class part 
opium poppy and class by class sorting processing based on the character. 
Adaptive processing is a thing of the following techniques. 

[0004]By namelythe thing for which the predicted value of the pixel of an original 
image is calculated by the linear combination of the pixel (suitably henceforth an 
input pixel) for which an inputted image (picture of the processing object of class 
sorting adaptation processing) is constituted from adaptive processingfor 
exampleand a predetermined prediction coefficient. The picture which removed the 
noise contained in the inputted imagethe picture which has improved the dotage 
produced in the inputted imagethe picture which raised resolutionetc. are acquired. 
[0005]While using as teacher data original images (for examplethe picture which 
does not contain a noisea picture without dotagea high-resolution pictureetc.) 
nowspecificallyfor exampleThe inputted image which superimposed the noise on 
the original imageor added the shading off is used as student dataThe predicted 
value E [y] of the pixel value y of the pixel (suitably henceforth original picture 
matter) which constitutes an original image A set of pixel value of some input 
pixels (pixel which constitutes an inputted image)x 2 and ...It considers asking by the 
primary linearity coupled models specified by the linear combination of 



predetermined prediction coefficient w^and ... In this casethe predicted value E 

[y] can be expressed with a following formula. 

[0006] 

E [y] =w 1 x 1 +w 2 x 2 + (1) 

Procession Y* which becomes by the procession X which becomes by the 
procession W which becomes by set of the prediction coefficient wand the student 
data aggregate in order to generalize a formula (1)and set of predicted value E 
[y][0007] 
[Equation 1] 

If a definition is come out and giventhe following observation equations will be 

materialized. 

[0008] 

XW=Y f ... (2) 

In [ ingredient x y of the procession X means here the j-th student data in the 
student data aggregate (student data aggregate used for prediction of teacher 
data yj of eye i affair) of eye i affairand ] a formula (1) ingredient Wj of the 
procession WThe prediction coefficient which a product with the j-th student data 
in the student data aggregate calculates is expressed, y, expresses the teacher 
data of eye i affairtherefore E [yj expresses the predicted value of the teacher 
data of eye i affair. 

[0009]And it considers applying a least square method to this observation 
equationand calculating the predicted value E [y] near the pixel value y of original 
picture matter. In this casethe procession E which becomes by set of the 
remainder e of predicted value E [y] to the procession Y which becomes by set of 
the true pixel value (true value) y of the original picture matter used as teacher 
dataand the pixel value y of original picture matter [0010] 
[Equation 2] 

If a definition is come out and giventhe following remainder equations will be 

materialized from an equation (2). 

[0011] 

XW=Y+E ... (3) 

In this caseprediction coefficient Wj for calculating the predicted value E [y] near 
the pixel value y of original picture matter is a square error. [0012] 
[Equation 3] 

It can ask by using the minimum. 

[001 3]Thereforewhen what differentiated the above-mentioned square error from 
prediction coefficient W; is set to Oin order that prediction coefficient Wj which fills 
a following formula may calculate the predicted value E [y] near the pixel value y 



of original picture matterit will call it an optimum value. 

[0014] 

[Equation 4] 



... (4) 

Thena following formula is first materialized by differentiating a formula (3) from 
prediction coefficient Wj. 
[0015] 
[Equation 5] 



... (5) 

A formula (6) is obtained from the formula (4) and (5). 

[0016] 

[Equation 6] 



... (6) 

If the student data x in the remainder equation of an equation (3)the prediction 
coefficient wteacher data yand the relation of the remainder e are taken into 
considerationthe following normal equations can be obtained from an equation (6). 
[0017] 
[Equation 7] 

... (7) 

Each equation which constitutes the normal equation of an equation (7) is that 
only a certain amount of number prepares the student data x and teacher data 
yThe optimal prediction coefficient w can be calculated by being able to build only 
the same number as the number of the prediction coefficients w which should be 
calculatedtherefore solving a formula (7) (howeverin order to solve a formula (7)in 
a formula (7Xhe procession which comprises a coefficient concerning the 
prediction coefficient w needs to be Masanori). In solving a formula (7)it is possible 
to sweep out and touse law (elimination of Gauss-Jordan) etc. for example. 
[001 8] Adaptive processing calculates the optimal prediction coefficient w as 
mentioned aboveand calculates the predicted value E [y] near the pixel value y of 
original picture matter by a formula (1) further using the prediction coefficient w. 
[001 9] Although adaptive processing is not included in an inputted imageit is the 
point that the ingredient contained in an original image is reproducedand differs 
from mere interpolation processingfor example. Namelyin adaptive processingas 
long as a formula (1) is seenit is the same as that of the interpolation processing 
what is called using an interpolation filterbut. Since [ for which the prediction 
coefficient w equivalent to the tap coefficient of the interpolation filter uses 



teacher data y ] it asks by study so to speakthe ingredient contained in an original 
image is reproducible. As adaptive processing can be called processing whichso to 
speakhas a creation (resolution imagination) operation of a picture and being 
mentioned above from thisThe predicted value of the original image which removed 
a noise and dotage from the inputted image is calculatedand also when changing 
the picture of a low resolution or standard resolution into the picture of high 
resolutionit can usefor example. 
[0020] 

[Problem(s) to be Solved by the Invention]As mentioned abovein class sorting 
adaptation processingadaptive processing is performed for every classbut. Original 
picture matter which is going to calculate a predicted value in the class sorting 
performed in the preceding paragraph (original picture matter) Since it does not 
necessarily actually exist other than the time of studyherelt observes for being 
assumingtwo or more input pixels around the position of the noticed picture 
element are extractedand the class division of the noticed picture element is 
carried out based on the character (for examplethe pattern of the pixel value 
which are two or more of the input pixelsthe inclination of a pixel valueetc). And 
as two or more input pixels used for this class sortingit sees from a noticed 
picture element and the input pixel in a fixed position is extracted. 
[0021]Howeverin the case where an inputted image with dotage is changed into 
the picture which has improved the dotage by class sorting adaptation 
processingfor example etc. Regardless of the degree of dotage of the inputted 
imageit sees from a noticed picture elementand if the input pixel in a fixed position 
was used for the class sorting of the noticed picture elementclass part [ which 
fully reflected the character of the noticed picture element ] being injured may 
become difficult. 

[0022]That iswhenperforming class sorting adaptation processingfor an inputted 
image with the small degree of dotage (degree of dotage) for exampleit can see 
from a noticed picture element and the direction which performs class sorting 
using the input pixel in a comparatively near position can perform the class 
division which fully reflected the character of the noticed picture element. When 
performing class sorting adaptation processing for an inputted image with the large 
degree of dotageit can see from a noticed picture element and the direction which 
performs class sorting using the input pixel in a comparatively far position can 
perform the class division which fully reflected the character. 

[0023]For examplewhen the inputted image of the peripheral part of the position of 
a noticed picture element is flatthe class division in which the direction which 
performs class sorting using the input pixel in the wide range of the inputted image 
in which a certain amount of change is seen fully reflected the character of the 
noticed picture element can be performed. 

[0024]Thereforesee from a noticed picture elementand if the input pixel in a fixed 
position was used for the class sorting of the noticed picture elementThe picture 
which has fully improved the inputted image may not be acquired according to the 
thing which fully reflected the character of the noticed picture element and which 



it cannot be injured by a class and the processing performance of class sorting 
adaptation processing deteriorates as a resulti.e. class sorting processing. 
[0025]Although it sees from a noticed picture elementand the linear prediction 
type of a formula (1) calculates and the predicted value which is a noticed picture 
element is calculated also by adaptive processing using the input pixel in a fixed 
positionlt is expected that see from a noticed picture element and the small 
predicted value of a prediction error [ as opposed to a noticed picture element in 
the direction which calculated the linear prediction type of the formula (1) ] is 
calculated like the case in class sorting also in this case using the input pixel in a 
variable position if needed. 

[0026]This invention is made in view of such a situationfor exampleit enables it to 
raise processing performancessuch as class sorting adaptation processing. 
[0027] 

[Means for Solving the Problem]This invention is characterized by the 1st data 
processing device comprising the following. 

A determination means to determine two or more data extracted from input data 
based on the waveform characteristic of input data. 

An extraction means to extract two or more data from input data about attention 
output data which is output data which is going to calculate a predicted value 
according to determination in a determination means. 

A prediction means which calculates a predicted value of attention output data 
based on two or more data extracted in an extraction means. 

[0028]In the 1st data processing devicebased on two or more data extracted in an 
extraction meansA class sorting means to perform class sorting about attention 
output dataand to output a corresponding class code can be formed furtherand a 
prediction means can be made to be asked for a predicted value of attention 
output data in this case using a predetermined prediction coefficient 
corresponding to a class code. 

[0029]A prediction means can be made to linearity-primary-predict attention 
output data using a prediction coefficient. A prediction means can be made to 
linearity-primary-predict attention output data using a prediction coefficient and 
two or more data extracted in an extraction means. 

[0030]A prediction coefficient memory measure which has memorized a prediction 
coefficient can be further provided in the 1st data processing device for every 
class code. 

[0031 ]A determination means can be made to be able to detect an extremum of 
input dataand two or more data extracted from input data can be made to 
determine it as it based on the detection result. A determination means can be 
made to be able to calculate a difference value of input dataand two or more data 
extracted from input data can be made to determine it as it based on the result of 
an operation. A determination means can be made to be able to calculate an error 
of input data to a function which approximates input dataand two or more data 
extracted from input data can be made to determine it as it based on the result of 



an operation. 

[0032]Input data and output data can be made into image data. An extraction 
means can be made to extract a pixel which exists on the outskirts spatially or in 
time from image data as input data to a pixel as attention output data in this case. 
[0033]This invention is characterized by the 1st data processing method 
comprising the following. 

A determination step which determines two or more data extracted from input 

data based on the waveform characteristic of input data. 

An extraction step which extracts two or more data from input data about 

attention output data which is output data which is going to calculate a predicted 

value according to determination in a determination step. 

A prediction step which calculates a predicted value of attention output data 

based on two or more data extracted in an extraction step. 

[0034]This invention is characterized by a program which a computer is made to 
execute comprising the following in the 1st medium. 

A determination step which determines two or more data extracted from input 

data based on the waveform characteristic of input data. 

An extraction step which extracts two or more data from input data about 

attention output data which is output data which is going to calculate a predicted 

value according to determination in a determination step. 

A prediction step which calculates a predicted value of attention output data 

based on two or more data extracted in an extraction step. 

[0035]This invention is characterized by the 2nd data processing device 
comprising the following. 

A creating means which generates student data which serves as a student from 

teacher data which serves as a teacher for study of a prediction coefficient. 

A determination means to determine two or more data extracted from student 

data based on the waveform characteristic of student data. 

An extraction means to extract two or more data from student data about 

attention teacher data which is teacher data which is going to calculate a 

predicted value according to determination in a determination means. 

A calculating means which calculates a prediction coefficient based on two or 

more data extracted in an extraction means. 

[0036]In the 2nd data processing devicebased on two or more data extracted in an 
extraction meansA class sorting means to perform class sorting about attention 
teacher dataand to output a corresponding class code can be formed furtherand a 
calculating means can be made to be asked for a prediction coefficient for every 
class code in this case using two or more data extracted in an extraction means. 
[0037]A calculating means can be made to be asked for a prediction coefficient for 
teacher data to be obtained by linearity primary prediction. A calculating means 
can be made to be asked for a prediction coefficient for teacher data to be 



obtained by linearity primary prediction using two or more data extracted in an 
extraction means. 

[0038]A determination means can be made to be able to detect an extremum of 
student dataand two or more data extracted from student data can be made to 
determine it as it based on the detection result. A determination means can be 
made to be able to calculate a difference value of student dataand two or more 
data extracted from student data can be made to determine it as it based on the 
result of an operation. A determination means can be made to be able to calculate 
an error of student data to a function which approximates student dataand two or 
more data extracted from student data can be made to determine it as it based on 
the result of an operation. 

[0039]Student data and teacher data can be made into image data. In this casean 
extraction means can be made to extract a pixel which exists on the outskirts 
spatially or in time from image data as student data to a pixel as attention teacher 
data. 

[0040]This invention is characterized by the 2nd data processing method 
comprising the following. 

A generation step which generates student data which serves as a student from 

teacher data which serves as a teacher for study of a prediction coefficient. 

A determination step which determines two or more data extracted from student 

data based on the waveform characteristic of student data. 

An extraction step which extracts two or more data from student data about 

attention teacher data which is teacher data which is going to calculate a 

predicted value according to determination in a determination step. 

An arithmetic step which calculates a prediction coefficient based on two or more 

data extracted in an extraction step. 

[0041]This invention is characterized by a program which a computer is made to 
execute comprising the following in the 2nd medium. 

A generation step which generates student data which serves as a student from 

teacher data which serves as a teacher for study of a prediction coefficient. 

A determination step which determines two or more data extracted from student 

data based on the waveform characteristic of student data. 

An extraction step which extracts two or more data from student data about 

attention teacher data which is teacher data which is going to calculate a 

predicted value according to determination in a determination step. 

An arithmetic step which calculates a prediction coefficient based on two or more 

data extracted in an extraction step. 

[0042]This invention is characterized by the 3rd data processing device comprising 
the following. 

The 1 st determination means that determines two or more data extracted from 

input data based on the waveform characteristic of input data. 

The 1st extraction means that extracts two or more 1st data from input data 



about attention output data which is output data which is going to calculate a 
predicted value according to determination in the 1st determination means. 
A prediction means which calculates a predicted value of attention output data 
based on two or more 1st data extracted in the 1st extraction means. 
A creating means which generates student data which serves as a student from 
teacher data which serves as a teacher for study of a prediction coefficientThe 
2nd determination means that determines two or more data extracted from 
student data based on the waveform characteristic of student dataA calculating 
means which calculates a prediction coefficient about attention teacher data 
which is teacher data which is going to calculate a predicted value based on two 
or more data extracted in the 2nd extraction means that extracts two or more 2nd 
data from student dataand the 2nd extraction means according to determination in 
the 2nd determination means. 

[0043]In the 1st data processing device of this inventiona data processing 
methodand a mediumBased on the waveform characteristic of input datatwo or 
more data extracted from input data is determinedand two or more data is 
extracted from input data about attention output data which is output data which 
is going to calculate a predicted value according to the determination. And a 
predicted value of attention output data is calculated based on two or more of the 
extracted data. 

[0044]In the 2nd data processing device of this inventiona data processing 
methodand a mediumstudent data which serves as a student is generated from 
teacher data which serves as a teacher for study of a prediction coefficientand 
two or more data extracted from student data is determined based on the 
waveform characteristic of the student data. And about attention teacher data 
which is teacher data which is going to calculate a predicted value according to 
the determinationtwo or more data is extracted from student dataand a prediction 
coefficient is calculated based on two or more of the extracted data. 
[0045]In the 3rd data processing device of this inventionbased on the waveform 
characteristic of input datatwo or more data extracted from input data is 
determinedand two or more 1st data is extracted from input data about attention 
output data which is output data which is going to calculate a predicted value 
according to the determination. And a predicted value of attention output data is 
calculated based on two or more of the 1st extracted data. On the other 
handstudent data which serves as a student is generated from teacher data which 
serves as a teacher for study of a prediction coefficientand two or more data 
extracted from student data is determined based on the waveform characteristic 
of the student data. And about attention teacher data which is teacher data which 
is going to calculate a predicted value according to the determinationtwo or more 
2nd data is extracted from student dataand a prediction coefficient is calculated 
based on two or more of the extracted data. 
[0046] 

[Embodiment of the Invention] Drawing 1 shows the example of composition of the 



1 embodiment of the image processing device which applied this invention. 
[0047]In this image processing deviceif SD (Standard Density) picture with low 
resolution is inputted as an inputted image for exampleby performing class sorting 
adaptation processing to that inputted imageThe picture (HD (High Density) 
picture) of high resolution is outputted. 

[0048]Namelythis image processing device comprises frame memory 1 class tap 
generating circuit 2prediction tap generating circuit 3class sorting circuit 4and 
coefficient RAM(Random Access Memory) 5the prediction arithmetic circuit 6and 
the tap determining circuit 7and thereThe inputted image (SD picture) used as the 
object which improves resolution is inputted. 

[0049]The frame memory 1 is made as [ store /for example / the inputted image 
inputted into an image processing device / temporarily / per frame ]. At this 
embodimentthe frame memory 1 can perform the processing in real timeeven if the 
inputted image which can memorize the inputted image of a multiple frame now 
with a bank changeand is inputted into an image processing device by this is an 
animation. 

[0050]Original picture matter which the class tap generating circuit 2 tends to ask 
for a predicted value by class sorting adaptation processing (here) The input pixel 
which uses the pixel of a high-resolution HD image for the class sorting about the 
noticed picture element as a noticed picture elementAccording to the tap 
information from the tap determining circuit 7it extracts from the inputted image 
memorized by the frame memory land this is outputted to the class sorting circuit 
4 as a class tap. 

[0051 ]The input pixel used for the prediction tap generating circuit 3 calculating 
the predicted value of a noticed picture element in the prediction arithmetic 
circuit 6According to the tap information from the tap determining circuit 7it 
extracts from the inputted image memorized by the frame memory land is made 
as [ supply / the prediction arithmetic circuit 6 ] by making this into a prediction 
tap. 

[0052]Based on the class tap from the class tap generating circuit 2the class 
sorting circuit 4 carries out class sorting of the noticed picture elementand is 
made as [ give / to coefficient RAMS / the class code corresponding to the class 
obtained as a result / as an address ]. Namelythe class sorting circuit 4 the class 
tap from the class tap generating circuit 2For example! -bit ADRC (Adaptive 
Dynamic Range Coding) processing is carried outand the ADRC code obtained as 
a result is outputted to coefficient RAMS as a class code. 

[0053]In K bit ADRC processingthe maximum MAX and the minimum MIN of a pixel 
value of an input pixel which constitute a class tap are detected herefor 
exampleDR=MAX-MIN is used as the local dynamic range of a setand re 
quantization of the input pixel which constitutes a class tap is carried out to K bit 
based on this dynamic range DR. That isout of the pixel value of the pixel which 
constitutes a class tapthe minimum MIN is subtracted and division (quantization) 
of the subtraction value is done by DR/2 K . Thereforewhen 1-bit ADRC processing 
of the class tap is carried outthe pixel value of each input pixel which constitutes 



the class tap will be 1 bit. And the bit string which put in order the pixel value of 1 
bit about each pixel which constitutes a class tap acquired by making it above in 
this case in predetermined order is outputted as an ADRC code. 
[0054]If coefficient RAMS has memorized the prediction coefficient for every class 
obtained by performing study in the learning device mentioned later and a class 
code is supplied from the class sorting circuit 4The prediction coefficient 
memorized to the address corresponding to the class code is readand the 
prediction arithmetic circuit 6 is supplied. 

[0055]The prediction coefficient w about the class of a noticed picture element to 
which the prediction arithmetic circuit 6 is supplied from coefficient 
RAM5w 2 and ...By performing the operation shown in the formula (1) using 
prediction tap (pixel value of each pixel to constitute) from the prediction tap 
generating circuit 3x 2 and ...the predicted value E [y] of the noticed picture element 
y is calculatedand this is outputted as a pixel value of the pixel which has 
improved resolution. 

[0056]Based on the waveform characteristic of the inputted image memorized by 
the frame memory 1the tap determining circuit 7The input pixel which makes a 
class tap and a prediction tap constitute is determinedand the information 
(suitably henceforth tap information) about the input pixel which makes the class 
tap and prediction tap constitute is supplied to the class tap generating circuit 2 
and the prediction tap generating circuit 3. 

[0057]That isthe tap determining circuit 7 is a central pixel (here) about the input 
pixel in the position of a noticed picture elementfor exampleas a dotted line 
surrounds and shows to drawing 2 . For examplesquare-like the class tap and 
prediction tap which horizontal x length becomes at 5x5 pixels as meaning the 
pixel in the position of the center of gravityfor example. It assumes as a 
fundamental tap (suitably henceforth a basic tap) which makes a class tap 
generating circuit and the prediction tap generating circuit 3 constitute (both are 
summarized and it is only hereafter called a tap suitably). 
[0058]Herein drawing 2 x seal shows the pixel (HD picture element) which 
constitutes an HD imageand O seal shows the pixel (SD pixel) which constitutes 
SD picture as an inputted image. Thereforein drawing 2 SD picture is what set the 
pixel number of the side of an HD imageand each length to one half. 
[0059]If a basic tap is assumedthe tap determining circuit 7 will detect the 
direction which passes along other SD pixels (input pixel) of a basic tap by making 
the central pixel into the starting pointfor exampleas shown in drawing 3 . Hereas a 
basic tap is mentioned abovewhen it comprises 5x5 pixelsas shown in drawing 
3direction of 16 thru/or D 16 which makes a central pixel the starting point will 
be detected. 

[0060]The tap determining circuit 7 determines SD pixel which makes a tap 
constitute eventually based on the waveform characteristic (characteristic of a 
pixel value) of SD pixel which exists in the direction about Dfor all directions } 
(here i= 12.. .16). That isabout Dfor all directions jbased on the waveform 
characteristic of SD picture which exists in the directionas shown in drawing 4t he 



position of SD pixel which constitutes the tap which exists in the direction is 
moved. And the tap determining circuit 7 supplies the information about the 
position of SD pixel which will constitute a tap eventually as tap information to the 
class tap generating circuit 2 and the prediction tap generating circuit 3 as a 
result of the movement. 

[0061]Nextwith reference to the flow chart of drawing 5 the resolution 
improvement processing performed in the image processing device of drawing 1 in 
which the resolution of an inputted image is improved is explained. 
[0062]To the frame memory 1SD picture (video) which is an object of resolution 
improvement processing is supplied one by one per frame as an inputted imageand 
the inputted image supplied per frame such is memorized one by one in the frame 
memory 1. 

[0063]And the predetermined HD picture element which is going to calculate a 
predicted value is made into a noticed picture elementand in Step S1 the tap 
determining circuit 7Based on the waveform characteristic of the inputted image 
memorized by the frame memory 1the input pixel which makes the tap about a 
noticed picture element constitute is determinedand the tap information about the 
position of the input pixel is outputted to the class tap generating circuit 2 and the 
prediction tap generating circuit 3. 

[0064]The class tap generating circuit 2 or the prediction tap generating circuit 
3In [ if tap information is received from the tap determining circuit 7 ] Step 
S2According to the tap informationthe input pixel which makes the class tap or 
prediction tap about a noticed picture element which is going to calculate a 
predicted value constitute is read from the frame memory land this constitutes a 
class tap or a prediction taprespectively. This class tap or prediction tap is 
supplied to the class sorting circuit 4 or the prediction arithmetic circuit 
6respectively. 

[0065]If a class tap is received from the class tap generating circuit 2in Step 
S3based on the class tapthe class sorting circuit 4 will perform class sortingand 
will output as an address the class code obtained as a result to coefficient RAMS 
about a noticed picture element. In step S4coefficient RAMS reads the prediction 
coefficient memorized to the address corresponding to the class code from the 
class sorting circuit 4and it supplies it to the prediction arithmetic circuit 6. 
[0066]In Step SSby performing the operation of the linear prediction type showing 
in a formula (1) using the prediction tap from the prediction tap generating circuit 
3and the prediction coefficient from coefficient RAMSthe predicted value E [y] of 
the noticed picture element (HD picture element) y is calculatedand it progresses 
to Step S6 in the prediction arithmetic circuit 6. In Step S6the prediction 
coefficient arithmetic circuit 6 outputs the predicted value E [y] of the noticed 
picture element y for which it asked at Step S5 as an HD picture element which 
has improved resolutionand it progresses to Step S7. 

[0067]It is judged whether the predicted value of all the HD picture elements 
which constitute the frame of a noticed picture element from Step S7 was 
calculatedWhen judged with not having asked yetit returns to Step S1 and the 



processing after Step S1 is newly repeated as a noticed picture element in the HD 
picture element which has not been asked for a predicted value yet. 
[0068]When judged with the predicted value of all the HD picture elements which 
constitute the frame of a noticed picture element having been calculated in Step 
S7It progresses to Step S8 and it is judged whether the inputted image (SD 
picture) corresponding to the frame of the HD image which should be processed 
next is memorized by the frame memory 1. In Step S8the inputted image 
corresponding to the frame of the HD image which should be processed nextWhen 
judged with the frame memory 1 memorizingin the predetermined HD picture 
element of the frame of the HD image which should be processed to the nextas a 
noticed picture elementit returns to Step S1 and the same processing is newly 
repeated hereafter. 

[0069]On the other handin Step S8when the inputted image corresponding to the 
frame of the HD image which should be processed next is judged as the frame 
memory 1 not memorizingresolution improvement processing is ended. 
[0070]Nextwith reference to the flow chart of drawing 6 a 1 st embodiment of the 
tap decision processing in Step S1 of drawing 5 performed in the tap determining 
circuit 7 is described. 

[0071]It is hereafter called a tap suitably one by one [ its / pixel ] besides the 
whole set of SD pixel used for the operation of class sorting and the linear 
prediction type of a formula (1). 

[0072]In the tap determining circuit 7a basic tap is set up in Step S1 1. Namelyin 
Step S1 1as the dotted line surrounded and showed to drawing 2 the tap which 
horizontal x length becomes at 5x5 pixels is set up as a basic tap by making the 
input pixel in the position of a noticed picture element into a central pixel. 
[0073]And it progresses to Step S12and as shown in drawing 3 direction of 16 
thru/or D 16 which passes along other SD pixels of a basic tap is detected by 
making a central pixel into the starting pointand it progresses to Step S13. 
[0074]The variable i which expresses with Step S13 direction of 16 thru/or D 16 
detected at Step S12 is initialized by 1for exampleand progresses to Step S14and 
the variable j showing the tap on Dfor all directions j is initialized by 1for 
exampleand progresses to Step S15. 

[0075]In Step S15it is judged whether SD pixel (pixel value) which is tap T MntKetHh 
andj from the central pixel on direction Dj serves as an extremum. When judged with 
SD pixel used as tap T iandj not serving as an extremum in Step S15progress to 
Step S16 and Tap T; on direction Diandj The tap outside it (direction which is not a central 
pixel side) is moved to the position which separated only 1 pixel from the central 
pixelfor example. And it returns to Step S15 and the same processing is repeated 
hereafter. 

[0076]In Step S15when judged with SD pixel used as tap T iandj serving as an 
extremumit progresses to Step S17and it ************** s only 1 and the 
variable j progresses to Step S18. It is judged whether it is below pixel number J. 
which constitutes taps other than a central pixel which have the variable j on 
direction Dj from Step S18. 



[0077]Herefor example in drawing 3 direction D^Dg...^ about D 15 are 2and 
direction D 2 D 4 D 6 ...Jj about D 16 are 1. 

[0078]When judged with it being below pixel number Jj which constitutes taps 
other than a central pixel which have the variable j on direction D; in Step S18That 
iswhen all the SD pixels that constitute taps other than a central pixel on direction 
Dj have not been supposed that the extremum is taken yetit returns to Step S15 
and the same processing is repeated for tap T iinttBPMlti ^ rfSDpix- ^ iohiBnrtfte .^ rauMndj . 
[0079]. In Step S18the variable j is on direction Dj. When judged with it not being 
below pixel number J- which constitutes taps other than a central pixel (i.e.when all 
the SD pixels that constitute taps other than a central pixel on direction Dj are 
presupposed that the extremum is taken)it progresses to Step S19 and 
************** s the variable i only for 1. And it progresses to Step S20 and it is 
judged whether the variable i is below the total I of the direction detected at Step 
S12 (the embodiment of drawing 3 16). 

[0080]In Step S20when judged with the variable i being below I (i.e.when SD pixel 
used as an extremum is used as the tap yet about no directions detected at Step 
S12)it returns to Step S14 and the same processing is repeated hereafter. 
[0081]When it is judged with the variable i not being below I in Step S20 on the 
other handNamelywhen SD pixel used as an extremum is used as a tap about all 
the directions detected at Step S12progress to Step S21 and the information 
about the position of SD pixel which constitutes the tap now as tap informationlt 
is outputted to the class tap generating circuit 2 and the prediction tap generating 
circuit 3and a return is carried out. 

[0082]According to a 1st embodiment of the above tap decision processingsin the 
class tap generating circuit 2 and the prediction tap generating circuit 3a tap as 
shown in drawing 7 will be constituted. 

[0083]That is drawing 7 (also setting to drawing 9 and drawing 1 1 which are 
mentioned later the same) shows SD pixel in the direction and direction D 9 
opposite to it 180 degrees for examplepaying attention to direction D 1 of the 
directions of 16 shown in drawing 3 . 

[0084]In drawing 7 (also setting to drawing 9 and drawing 1 1 the same). While 
expressing with n the position of the central pixel which constitutes a tap and 
expressing with n-m further the position of the pixel which only m pixel (m is a 
positive integer) has in the direction of D 1 from the position nthe position of the 
pixel which only m pixel has in the direction of D 9 from the position n is expressed 
with n+m. 

[0085]In drawing 7 (also setting to drawing 9 and drawing 1 1 t he same)x seal 
shows SD pixel (pixel value)and O seal shows the tap (SD pixel which has become). 
[0086]Two SD pixels by which two SD pixels which adjoin in the direction about 
direction D, adjoin in the direction also about direction D 9 again serve as a tap to 
SD pixel which drawing 7 (A) shows the basic taptherefore is a central pixel. 
[0087] Drawing 7 (B) to SD pixel which shows the tap constituted by the tap 
decision processing of drawing 6 and is a central pixel from the basic tap of drawing 
7_(A) about direction D v SD pixel which SD pixel used as the maximal value which 



appears first in accordance with the directionand SD pixel used as the minimal 
value serve as a tapand serves as the minimal value first expressed in accordance 
with the direction also about direction D 9 and SD pixel used as the maximal value 
serve as a tap. 

[0088]Nextwith reference to the flow chart of drawing 8 a 2nd embodiment of the 
tap decision processing in Step S1 of drawing 5 performed in the tap determining 
circuit 7 is described. 

[0089]In Step S31 or S32in the tap determining circuit 7the respectively same 
processing as the case in Step S1 1 or S12 of drawing 6 is performedand by 
thisDetection of direction of 16 thru/or D 16 which passes along SD pixel of 
setting out of a basic tap and everything but the basic tap which makes a central 
pixel the starting point is performed. 

[0090]And progress to Step S33 and like the case in Step S13 of drawing 6 The 
variable i showing the direction detected at Step S32 is initialized by 1for 
exampleand progresses to Step S34and dynamic range DR; of SD pixel used as the 
tap on direction Dj calculates. That isin Step S34the maximum and the minimum 
(the greatest pixel value and the minimum pixel value) are detected out of all (a 
central pixel is included) SD pixels used as the tap on direction D^nd the 
difference of the maximum and minimum is called for as dynamic range DR;. 
[0091] After calculation of dynamic range DRjit progresses to Step S35 and it is 
judged from the threshold th to which the dynamic range DRj was set beforehand 
whether it is size. When judged with dynamic range DRj being below the threshold 
th in Step S35among SD pixels used as the tap on direction DjWhen there is no so 
big changetherefore stationarity is observed in the tap on direction Djit progresses 
to Step S36and SD pixel which constitutes a tapexcluding a central pixelis 
changed so that the interval of the taps on direction Dj may spread by 1 pixelfor 
example. And it progresses to Step S34 and the same processing is repeated 
hereafter. 

[0092]Among SD pixels which serve as a tap on direction Dj when judged with 
dynamic range DRj being size from the threshold th in Step S35When it is admited 
that there is a comparatively big changetherefore there is no stationarity in the 
tap on direction Djit progresses to Step S37 and **************s the variable i 
only 1. And it progresses to Step S38 and it is judged whether the variable i is 
below the total I of the direction detected at Step S32. 

[0093]When judged with the variable i being below I in Step S38That isabout all the 
directions detected at Step S32when SD pixel which a dynamic range consists 
large of from the threshold th has not been used as the tap yetit returns to Step 
S34 and the same processing is repeated hereafter. 

[0094]When it is judged with the variable i not being below I in Step S38 on the 
other handNamelywhen SD pixel which a dynamic range consists large of from the 
threshold th is used as a tap about all the directions detected at Step S32It 
progresses to Step S39and nowas tap informationthe information about the 
position of SD pixel which constitutes the tap is outputted to the class tap 
generating circuit 2 and the prediction tap generating circuit 3and carries out a 



return. 

[0095]According to a 2nd embodiment of the above tap decision processingsin the 
class tap generating circuit 2 and the prediction tap generating circuit 3a tap as 
shown in drawing 9 will be constituted. 

[0096]Namely drawing 9 (A) to SD pixel which shows the basic taptherefore is a 
central pixel like drawing 7 (A) about direction Two SD pixels by which two SD 
pixels which adjoin in the direction adjoin in the direction also about direction D 9 
again serve as a tap. 

[0097] Drawing 9 (B) is shown from the basic tap of drawing 9 (A)and the tap 
constituted by the tap decision processing of drawing 8 about both direction 
and D 9 . SD pixel which makes dynamic range DR } and DR 9 larger than the 
threshold th serves as a tap. 

[0098]In the embodiment of drawing 8 and drawing 9 extended the interval of SD 
pixels used as a tap so that dynamic range DR; which is the difference of the 
maximum of the SD pixels used as the tap on direction D; and the minimum might 
become large from the threshold thbut. In additionit is possible to extend the 
interval of SD pixels used as a tap etc. so that the difference of what SD pixel 
which serves as a tap on direction Dfor example adjoins may become large from 
the threshold th. 

[0099]Nextwith reference to the flow chart of drawing 1 0 a 3rd embodiment of the 
tap decision processing in Step S1 of drawing 5 performed in the tap determining 
circuit 7 is described. 

[0100]In the tap determining circuit 7the respectively same processing as the case 
in Steps S11 thru/or S13 of drawing 6 is performed in Steps S41 thru/or S43. 
Therebydetection of direction of 16 which passes along other SD pixels of the 
basic tap which makes the starting point setting out of a basic tap and a central 
pixel thru/or D 16 and initialization showing the detected direction of the variable i 
are performed. 

[01 01] And it progresses to Step S44and for examplestraight-line (suitably 
henceforth approximation straight lines) L; as a function which approximates SD 
pixel (pixel value) used as the basic tap on direction D { is called forand it 
progresses to Step S45. SD pixel (a central pixel is included) which serves as a 
tap on direction D { in Step S45 — the error (for examplesquare error) over each 
approximation-straight-lines L ( is searched forand the integrated value of the error 
is calculated further. And it progresses to Step S46 and it is judged whether the 
integrated value of the error is size from a predetermined threshold. 
[0102]When it judges that an integrated value with error is not size from a 
predetermined threshold in Step S46among SD pixels used as the tap on direction 
DjWhen there is no so big change and stationarity is acceptedit progresses to Step 
S47 and all taps other than a central pixel on direction Dj are moved by 1 pixel 
along with the direction Dj (on outside). And it progresses to Step S45 and the 
same processing is repeated hereafter. 

[0103]Among SD pixels which serve as a tap on direction Dj when judged with an 
integrated value with error being size from a predetermined threshold in Step 



S46When there is a comparatively big change and stationarity is not acceptedit 
progresses to Step S48 and **************s the variable i only 1 . And it 
progresses to Step S49 and it is judged whether the variable i is below the total I 
of the direction detected at Step S42. 

[0104]In Step S49when judged with the variable i being below Inamelywhen SD 
pixel which an integrated value with error consists large of from a predetermined 
threshold is used as the tap yet about no directions detected at Step S42it 
returns to Step S44 and the same processing is repeated hereafter. 
[0105]When it is judged with the variable i not being below I in Step S49 on the 
other handNamelywhen SD pixel which an integrated value with error consists 
large of from a predetermined threshold about all the directions detected at Step 
S42 is used as a taplt progresses to Step S50and as tap informationthe 
information about the position of SD pixel which constitutes the tap now is 
outputted to the class tap generating circuit 2 and the prediction tap generating 
circuit 3and carries out a return. 

[0106]According to a 3rd embodiment of the above tap decision processingsin the 
class tap generating circuit 2 and the prediction tap generating circuit 3a tap as 
shown in drawing 1 1 will be constituted. 

[0107]Namely drawing 11 (A) to SD pixel which shows the basic taptherefore is a 
central pixel like drawing 7 (A) about direction D^ Two SD pixels by which two SD 
pixels which adjoin in the direction adjoin in the direction also about direction D 9 
again serve as a tap. 

[0108] Drawing 1 1 (B) is shown from the basic tap of drawing 1 1 (A)and the tap 
constituted by the tap decision processing of drawing 10 about both direction D } 
and D 9 . SD pixel which makes the integrated value of the error over 
approximation-straight-lines L } and L 9 larger than a predetermined threshold 
serves as a tap. 

[01 09] Although SD pixel to straight-line L s which approximates SD pixel used as a 
basic tap on direction Dj which makes a tap constitute based on the integrated 
value of the error of SD pixel which constitutes a tap was determined in the 
embodiment of drawing 10 and drawing 1 1 1n additionit is also possible todetermine 
SD pixel which makes a tap constitute for examplebased on distribution with error 
etc. 

[01 10]Although all taps other than a central pixel on direction D; were moved by 1 
pixel along with the direction D } in Step S47 in the embodiment of drawing 10 It 
may be made to move a tap in addition to thisin Step S47for example so that the 
interval of the taps on direction Dj may be extendedfor example by 1 pixel. 
[01 1 1] Although it asks for a straight line and the tap was determined in the 
embodiment of drawing 10 and drawing 1 1 based on the error over the straight line 
as a function which approximates SD pixel used as a basic tap on direction D}n 
additionit is also possible to search for the flat surface and curved surface which 
are a function which approximates SD pixel used as a basic tap which exists in the 
two or more directionsfor exampleand to determine a tap based on the error over 
the flat surface and curved surface. 



[01 1 2]Next drawing 12 shows the example of composition of the 1 embodiment of 
the learning device which calculates the prediction coefficient for every 
[ coefficient RAMS of drawing 1 is made to memorize ] class. 
[0113]The original image (HD image high-resolution here) used as teacher data y 
is supplied to the frame memory 61 per framefor exampleand the frame memory 
61 stores the original image temporarily at it. The infanticide circuit 62 generates 
SD picture with low resolution as student data by reading the original image set to 
teacher data y in the study of a prediction coefficient memorized by the frame 
memory 61 and thinning out the pixel number which constitutes the original image. 
The seal 3 and the infanticide circuit 62 generate SD pixel similarly shown in 
drawing 2 by O seal byfor examplethinning out the pixel number of the width and 
length of an HD image shown in drawing 2 by x seal to one halfrespectively. This 
SD picture is supplied to the frame memory 63. 

[0114]The frame memory 63 stores temporarily SD picture from the infanticide 
circuit 62. The frame memories 61 and 63 are constituted like the frame memory 1 
of drawing 1 . 

[01 15]The class tap generating circuit 64 or the prediction tap generating circuit 
65SD pixel which constitutes SD picture memorized by the frame memory 63 is 
usedLike the class tap generating circuit 2 of drawing 1 or the prediction tap 
generating circuit 3According to the tap information from the tap determining 
circuit 72about the teacher data used as a noticed picture elementa class tap or a 
prediction tap is constituted and the class sorting circuit 66 or the adder circuit 
67 is suppliedrespectively. A prediction tap is supplied also to the adder circuit 67. 
[0116]The class sorting circuit 66 is constituted like the class sorting circuit 4 of 
drawing 1 Based on the class tap from the class tap generating circuit 64class 
sorting of the noticed picture element is carried outand a corresponding class 
code is given as an address to the prediction tap memory 68 and the teacher data 
memory 70. 

[0117]The adder circuit 67 reads the memory value of the address corresponding 
to the class code which the class sorting circuit 66 outputs from the prediction 
tap memory 68and The memory valueThe operation equivalent to the summation 
(sigma) which serves as a multiplier of the prediction coefficient w in the left side 
of the normal equation of an equation (7) by adding SD pixel (pixel value) which 
constitutes the prediction tap from the prediction tap generating circuit 65 is 
performed. And the adder circuit 67 is made to memorize in the form which 
overwrites the result of an operation to the address corresponding to the class 
code which the class sorting circuit 66 outputs. 

[01 18]The prediction tap memory 68 memorizes the output value of the adder 
circuit 67 to the address while it reads the memory value of the address 
corresponding to the class which the class sorting circuit 66 outputs and supplies 
it to the adder circuit 67. 

[0119]While reading the noticed picture element of the original picture matter 
which constitutes the original image memorized by the frame memory 61 as 
teacher data ythe adder circuit 69By adding teacher data (original picture matter) 



y which read the memory value of the address corresponding to the class code 
which the class sorting circuit 66 outputs from the teacher data memory 70and 
read it from the memory value and the frame memory 61. The operation equivalent 
to the summation (sigma) in the right-hand side of the normal equation of an 
equation (7) is performed. And the adder circuit 69 is made to memorize in the 
form which overwrites the result of an operation to the address corresponding to 
the class code which the class sorting circuit 66 outputs. 

[0120]The multiplication in a formula (7) is also performed in the adder circuits 67 
and 69. That isin the adding machine 67the multiplication of the SD pixel x and 
teacher data y which the multiplication of SD pixel x which constitute a prediction 
tap is also performedand constitute a prediction tap from the adding machine 69 is 
also performed. Thereforealthough the SD pixel x which constitutes a prediction 
tap is needed in the adding machine 69this is supplied from the prediction tap 
generating circuit 65 as mentioned above. 

[0121]The teacher data memory 70 memorizes the output value of the adder 
circuit 69 to the address while it reads the memory value of the address 
corresponding to the class code which the class sorting circuit 66 outputs and 
supplies it to the adder circuit 69. 

[0122]the arithmetic circuit 71 — the prediction tap memory 68 or the teacher 
data memory 70 — from each. The prediction coefficient for every class is 
calculated by reading the memory value memorized to the address corresponding 
to each class code one by onebuilding the normal equation showing in an equation 
(7)and solving this, namelythe arithmetic circuit 71 — the prediction tap memory 
68 or the teacher data memory 70 — the prediction coefficient for every class is 
calculated by building the normal equation of an equation (7) and solving this from 
the memory value memorized to the address corresponding to each class code of 
each. 

[0123]The tap determining circuit 72 is performing the same tap decision 
processing as the tap determining circuit 7 of drawing 1 The tap information about 
the tap which the class tap generating circuit 64 and the prediction tap generating 
circuit 65 are made to generate is determinedand the class tap generating circuit 
64 and the prediction tap generating circuit 65 are supplied. 
[01 24]Nextwith reference to the flow chart of drawing 13 the learning processing 
which is performed in the learning device of drawing 1 2 and which calculates the 
prediction coefficient for every class is explained. 

[0125]The HD image as an original image (video) used as teacher data is supplied 
to a learning device per frameand the HD image is memorized one by one in the 
frame memory 61. And in Step S61the HD image as an original image memorized 
by the frame memory 61 is supplied to the infanticide circuit 62and let it be SD 
picture there. As student datafrom the infanticide circuit 62this SD picture is 
supplied to the frame memory 63and is memorized. 

[0126]And in [ the tap determining circuit 72 makes the predetermined HD picture 
element of the frame of the HD image corresponding to the frame of SD picture 
memorized by the frame memory 63 a noticed picture elementand ] Step S62SD 



pixel which makes the tap about the noticed picture element constitute is 
determined like the case in the tap determining circuit 7 of drawing l and the tap 
information about the SD pixel is outputted to the class tap generating circuit 64 
and the prediction tap generating circuit 65. 

[0127]In the class tap generating circuit 64 or the prediction tap generating circuit 
65. In Step S63according to the tap information from the tap determining circuit 
72SD pixel which makes the class tap or prediction tap about a noticed picture 
element constitute is read from the frame memory 63and a class tap or a 
prediction tap is constitutedrespectively. And a class tap is supplied to the class 
sorting circuit 66and a prediction tap is supplied to the adder circuits 67 and 69. 
[0128]In the class sorting circuit 66it is made to be the same as that of the case 
in the class sorting circuit 4 of drawing 1 in Step S64Class sorting of the noticed 
picture element is carried out using the class tap from the class tap generating 
circuit 64and the class code as the class sorting result is given as an address to 
the prediction tap memory 68 and the teacher data memory 70. 
[0129]And it progresses to Step S65 and an add lump of a prediction tap (student 
data) and teacher data is performed. 

[0130]Namelyin Step S65the prediction tap memory 68 reads the memory value of 
the address corresponding to the class code which the class sorting circuit 66 
outputsand supplies it to the adder circuit 67. The adder circuit 67 performs the 
operation equivalent to the summation (sigma) used as the multiplier of the 
prediction coefficient in the left side of the normal equation of an equation (7) 
using the memory value supplied from the prediction tap memory 68and SD pixel 
which constitutes a supplying-from prediction tap generating circuit 65 prediction 
tap. And the adder circuit 67 is made to memorize in the form which overwrites 
the result of an operation to the address of the prediction tap memory 68 
corresponding to the class code which the class sorting circuit 66 outputs. 
[01 31]In Step S65the teacher data memory 70 reads the memory value of the 
address corresponding to the class code which the class sorting circuit 66 
outputsand supplies it to the adder circuit 69. From the frame memory 61 the 
adder circuit 69 reads a noticed picture elementand performs the operation 
equivalent to the summation (sigma) in the right-hand side of the normal equation 
of an equation (7) using the HD picture element used as the read noticed picture 
elementa prediction tapand the memory value supplied from the teacher data 
memory 70. And the adder circuit 69 is made to memorize in the form which 
overwrites the result of an operation to the address of the teacher data memory 
70 corresponding to the class code which the class sorting circuit 66 outputs. 
[0132]Thenall the HD picture elements which progress to Step S66 and constitute 
the frame of a noticed picture elementWhen it is judged whether it processed as a 
noticed picture element and it is judged with not having carried out yetit returns to 
Step S62 and the same processing is newly hereafter repeated as a noticed 
picture element in the HD picture element which has not been made into the 
noticed picture element yet. 

[0133]In Step S66it is judged whether the frame of the HD image which should 



progress to Step S67 and then should process all the HD picture elements which 
constitute the frame of a noticed picture element when judged with having 
processed as a noticed picture element is memorized by the frame memory 61. In 
Step S67when the frame of the HD image which should be processed next is 
judged as the frame memory 61 memorizingit returns to Step S61 and the same 
processing is repeated for the frame of the HD image which should be processed 
to the next. 

[0134]The frame of the HD image which should be processed next in Step 
S67When judged with the frame memory 61 not memorizingnamelywhen it 
processes using all the HD images beforehand prepared for studyprogressing to 
Step S68 — the arithmetic circuit 71 — the prediction tap memory 68 or the 
teacher data memory 70 — from each. The prediction coefficient for every class 
is calculated by reading the memory value memorized to the address 
corresponding to each class code one by onebuilding the normal equation showing 
in an equation (7)and solving this. In Step S69the arithmetic circuit 71 outputs the 
prediction coefficient for the every class for which it askedand ends learning 
processing. 

[0135]In the learning processing of the above prediction coefficientsalthough the 
class from which a number required to calculate a prediction coefficient of normal 
equations are not obtained may ariseabout such a classit is able to make it to 
output a default prediction coefficient for exampleetc. 

[0136]As mentioned abovesince SD pixel which makes the tap constitute was 
determined based on the waveform characteristic of SD pixel which constitutes a 
tapFor examplethe composition of the class tap for performing the class division 
which fully reflected the character of the noticed picture elementand the 
prediction tap which makes the prediction error of a predicted value small can be 
attainedandas a resultthe processing performance of class sorting adaptation 
processing can be raised. 

[0137]Nexthardware can also perform a series of processings mentioned aboveand 
software can also perform. When software performs a series of processingsThe 
program which constitutes the software is installed in the computer built into the 
pixel processing unit and learning device as hardware for exclusive useor the 
general-purpose computer which performs various kinds of processings by 
installing various kinds of programs. 

[0138]Thenthe medium used in order to install in a computer the program which 
performs a series of processings mentioned above and to change it into the state 
which can be performed by computer with reference to drawing 14 is explained. 
[0139]As shown in drawing 14 (A)a user can be provided with a program in the 
state where it installed in the hard disk 102 and the semiconductor memory 103 as 
a recording medium which are built in the computer 101 beforehand. 
[0140]Or as shown in drawing 14 (B)again a programThe floppy disk 1 1 1 CD-ROM 
(Compact.) To recording mediasuch as Disc Read Only Memory112the MO 
(Magneto optical) disk 1 13DVD(Digital Versatile Disc) 114the magnetic disk 115and 
the semiconductor memory 1 1 6. It can store temporarily or permanently and can 



provide as a software package. 

[0141]Via [ as a program is shown in drawing 1 4 (C) ] the artificial satellite 122 for 
the digital satellite broadcasting from the download site 121 In [ transmit to the 
computer 101 on radioor transmit to the computer 123 with a cable via the 
networks 131 such as LAN (Local Area Network) and the Internetand ] the 
computer 101 It can be made to store in the hard disk 102 etc. to build in. 
[0142]The medium in this specification means the concept of a broad sense 
containing all these media. 

[0143]The step which describes the program provided by a medium in this 
specificationlt is not necessary to necessarily process to a time series in 
accordance with the order indicated as a flow chartand a parallel target or the 
processing (for exampleparallel processing or processing by an object) performed 
individually is also included. 

[0144]Next drawing 15 shows the example of composition of the computer 101 of 
drawing 14 . 

[0145]The computer 101 contains CPlKCentral Processing Unit) 142as shown in 
drawing 1 5 . Via the bus 141 the input/output interface 145 is connected to 
CPU142 and CPU142If instructions are inputted when the input part 147 which 
comprises a keyboarda mouseetc. is operated by the user via the input/output 
interface 145According to itthe program stored in ROM(Read Only Memory) 143 
corresponding to the semiconductor memory 103 of drawing 14 (A) is executed. Or 
a program by which CPU 142 is stored in the hard disk 102A program which was 
transmitted from the satellite 122 or the network 131 was received in the 
communications department 148and was installed on the hard disk 102Or the 
program which was read from the floppy disk 1 1 1 with which the drive 1 49 was 
equippedCD-ROM112MO disk 113DVD114or the magnetic disk 115and was 
installed on the hard disk 102It loads to RAM(Random Access Memory) 144and 
performs. And CPU 142 outputs the processing result to the indicator 146 which 
comprises LCD (Liquid CryStal Display) etc. via the input/output interface 145 if 
neededfor example. 

[0146]By linearity primary prediction class sorting application processing performs 
study which calculates a prediction coefficient for every class using teacher data 
and student dataand using the prediction coefficient and input data. Since the 
predicted value of teacher data is calculatedit becomes possible from input data 
to obtain the prediction coefficient for calculating a desired predicted value with 
the teacher data and student data which are used for study. That iswhile using a 
picture without dotage as teacher data for examplethe prediction coefficient which 
improves dotage can be obtained by using the picture which added dotage to the 
picture as student data. While using the picture which does not contain a noise as 
teacher data for examplethe prediction coefficient which removes a noise can be 
obtained by using the picture which added the noise to the picture as student data. 
Thereforethis invention can be appliedwhen improving dotagewhen [ other ] 
performing waveform equalizationfor exampleand when [ when removing a noise 
besides in the case of raising resolution which was mentioned above or ]. 



[0147]In this embodimentalthough video was set as the object of class sorting 
application processingit is possible to be further aimed at a still picturea soundthe 
signal (RF (RadioFrequency) signal) reproduced from the recording medium besides 
videoetc. 

[0148]In this embodimentalthough a class tap and a prediction tap will be 
constituted from being constituted according to the same tap information by the 
same pixellt is possible to have different composition from a class tap and a 
prediction tapi.e.also constitute according to different tap information. 
[01 49] Although both a class tap and a prediction tap are seen from a noticed 
picture element and SD pixel of the variable position constituted it from this 
embodiment according to tap informationlt sees from a noticed picture element 
and either a class tap or the prediction taps can be constituted from an SD pixel 
of a fixed position. 

[0150]Although SD pixel [ be / SD pixel which constitutes the tap / an 
extremum ] which makes a tap constitute based on the error over the dynamic 
rangeapproximation straight linesetc. was determined in this embodimentlt is 
possible to determine SD pixel which makes a tap constitute based on the feature 
(waveform characteristic) on the waveform of SD pixel which constitutes taps 
other than these. 

[0151]Although SD pixel which makes a tap constitute was determined in this 
embodiment for for [ every ] all directions which make a central pixel the starting 
pointit is not necessary to necessarily determine the pixel which makes a tap 
constitute for such every direction. 

[01 52]Although the image processing device and the learning device which learns 
the prediction coefficient for every class used with the image processing device 
were constituted from this embodiment as a separate deviceit is also possible to 
constitute these image processing devices and learning devices in one. And it is 
able to make it to make the prediction coefficient which makes learn to a learning 
device in real timeand is used for it with an image processing device update in real 
time in this case. 

[0153]At this embodimentalthough it was made to make coefficient RAM5 
memorize the prediction coefficient for every class beforehandthis prediction 
coefficient can also be made to supply an image processing device with an 
inputted image (SD picture) for example. 

[0154]A class tap and a prediction tap can also be constituted using the pixel 
which exists in the direction of spaceand which direction of a time direction. 
[0155]In learning processingas mentioned abovethe add lump equivalent to the 
summation (sigma) of a formula (7) is performed using a prediction tapbut. The add 
lump using the prediction tap which comprises an SD pixel of a position which 
sees and is different from a noticed picture element is performed for the pixels to 
which those prediction taps correspond. 

[0156]namelydirection of 16 shown in drawing 3 by this embodiment supposing 
it constituted the basic tap from 5x5 pixels 25 pixels centering on a central pixelas 
shown in drawing 3 t hru/or D 16 — a prediction tap comprises arbitrary pixels on 



each. Thereforealthough it comprises a pixel in a position which regards as a 
certain prediction tap and other prediction taps from a central pixeland is 
differentln this casefor examplein a certain prediction tapthe pixel as a tap which 
is in the 1st from the central pixel of direction D ] is added in other prediction taps 
with the pixel as a tap which is in the 1st from the central pixel of direction 
[0157]Class sorting is also performed from same viewpoint. Namelyeach pixel 
(pixel value) as a tap for all directions which constitute a certain class tapfor 
exampleWhen equal to each pixel as a tap in the turn of corresponding from the 
central pixel of a corresponding direction which constitutes other class tapsthe 
class sorting result of having used each of the two class taps becomes the same 
(class sorting is carried out to the same class). 
[0158] 

[Effect of the Invention]According to the 1st data processing device of this 
inventiona data processing methodand the mediumlike the above. Based on the 
waveform characteristic of input datatwo or more data extracted from input data 
is determinedand two or more data is extracted from input data about the 
attention output data which is output data which is going to calculate a predicted 
value according to the determination. And the predicted value of attention output 
data is calculated based on two or more of the extracted data. Thereforeit 
becomes possible to calculate the predicted value near attention output data. 
[0159]According to the 2nd data processing device of this inventiona data 
processing methodand the mediumthe student data which serves as a student is 
generated from the teacher data which serves as a teacher for study of a 
prediction coefficientand two or more data extracted from student data is 
determined based on the waveform characteristic of the student data. And about 
the attention teacher data which is teacher data which is going to calculate a 
predicted value according to the determinationtwo or more data is extracted from 
student dataand a prediction coefficient is calculated based on two or more of the 
extracted data. Thereforeit becomes possible to obtain the prediction coefficient 
which can calculate the predicted value near teacher data. 

[0160]According to the 3rd data processing device of this inventionbased on the 
waveform characteristic of input datatwo or more data extracted from input data 
is determinedand two or more 1st data is extracted from input data about the 
attention output data which is output data which is going to calculate a predicted 
value according to the determination. And the predicted value of attention output 
data is calculated based on two or more of the 1st extracted data. On the other 
handthe student data which serves as a student is generated from the teacher 
data which serves as a teacher for study of a prediction coefficientand two or 
more data extracted from student data is determined based on the waveform 
characteristic of the student data. And about the attention teacher data which is 
teacher data which is going to calculate a predicted value according to the 
determinationtwo or more 2nd data is extracted from student dataand a prediction 
coefficient is calculated based on two or more of the extracted data. Thereforeit 
becomes possible to calculate the predicted value near attention output data by 



using the prediction coefficient. 
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[Brief Description of the Drawings] 

[Drawing 1] It is a block diagram showing the example of composition of the 1 
embodiment of the image processing device which applied this invention. 
[Drawing 2] It is a figure showing a basic tap. 

[Drawing 3] It is a figure for explaining processing of the tap determining circuit 7 
of drawing 1 . 

[Drawing 4] It is a figure for explaining processing of the tap determining circuit 7 
of drawing 1 . 

[Drawing 5] It is a flow chart for explaining the resolution improvement processing 
by the image processing device of drawing 1 . 

[Drawing 6] It is a flow chart for explaining the 1st tap decision processing in Step 
S1 of drawing 5 . 

[Drawing 7] It is a figure for explaining the 1st tap decision processing in Step S1 
of drawing 5 . 

[Drawing 8] It is a flow chart for explaining the 2nd tap decision processing in Step 
S1 of drawing 5 . 

[Drawing 9] It is a figure for explaining the 2nd tap decision processing in Step S1 
of drawing 5 . 

[Drawing 10] It is a flow chart for explaining the 3rd tap decision processing in Step 
S1 of drawing 5 . 

[Drawing 1 1] It is a figure for explaining the 3rd tap decision processing in Step S1 
of drawing 5 . 

[Drawing 12] It is a block diagram showing the example of composition of the 1 
embodiment of the learning device which applied this invention. 
[Drawing 13] It is a flow chart for explaining the learning processing by the learning 
device of drawing 12 . 

[Drawing 14] It is a figure for explaining the medium which applied this invention. 
[Drawing 15] It is a block diagram showing the example of composition of the 
computer 101 of drawing 14 . 
[Description of Notations] 

1 A frame memory and 2 A class tap generating circuit3 A prediction tap 
generating circuit and 4 class sorting circuits5 coefficient RAM6 prediction 
arithmetic circuitsand 7 tap determining circuit61 A frame memory and 62 
infanticide circuits63 A frame memory and 64 A class tap generating circuit65 A 
prediction tap generating circuit and 66 class sorting circuits67 An adder circuit 
and 68 prediction-tap memory69 adder circuits and 70 Teacher data memory71 An 
arithmetic circuit and 72 A tap determining circuit101 computers and 102 A hard 
diskl 03 Semiconductor memory and 1 1 1 A floppy diskl 1 2 CD-ROM and 1 1 3 MO 
disks! 14 DVD115 magnetic disksand 116. Semiconductor memory and 121 A 



download site122 A satellite and 131 [An input part and 148 / The 
communications department and 149 / Drive ] A network and 141 A bus142 
CPU143 ROM144 RAMI 45 input/output interfacesand 146 An indicator and 147 



